{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MAGIC Gamma Telescope - TPOT Classification Study"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below gives information about the data set:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data are MC generated (see below) to simulate registration of high energy gamma particles in a ground-based atmospheric Cherenkov gamma telescope using the imaging technique. Cherenkov gamma telescope observes high energy gamma rays, taking advantage of the radiation emitted by charged particles produced inside the electromagnetic showers initiated by the gammas, and developing in the atmosphere. This Cherenkov radiation (of visible to UV wavelengths) leaks through the atmosphere and gets recorded in the detector, allowing reconstruction of the shower parameters. The available information consists of pulses left by the incoming Cherenkov photons on the photomultiplier tubes, arranged in a plane, the camera. Depending on the energy of the primary gamma, a total of few hundreds to some 10000 Cherenkov photons get collected, in patterns (called the shower image), allowing to discriminate statistically those caused by primary gammas (signal) from the images of hadronic showers initiated by cosmic rays in the upper atmosphere (background). \n",
    "\n",
    "Typically, the image of a shower after some pre-processing is an elongated cluster. Its long axis is oriented towards the camera center if the shower axis is parallel to the telescope's optical axis, i.e. if the telescope axis is directed towards a point source. A principal component analysis is performed in the camera plane, which results in a correlation axis and defines an ellipse. If the depositions were distributed as a bivariate Gaussian, this would be an equidensity ellipse. The characteristic parameters of this ellipse (often called Hillas parameters) are among the image parameters that can be used for discrimination. The energy depositions are typically asymmetric along the major axis, and this asymmetry can also be used in discrimination. There are, in addition, further discriminating characteristics, like the extent of the cluster in the image plane, or the total sum of depositions. \n",
    "\n",
    "https://archive.ics.uci.edu/ml/datasets/MAGIC+Gamma+Telescope\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Attribute Information:\n",
    "\n",
    "1. fLength: continuous # major axis of ellipse [mm] \n",
    "2. fWidth: continuous # minor axis of ellipse [mm] \n",
    "3. fSize: continuous # 10-log of sum of content of all pixels [in #phot] \n",
    "4. fConc: continuous # ratio of sum of two highest pixels over fSize [ratio] \n",
    "5. fConc1: continuous # ratio of highest pixel over fSize [ratio] \n",
    "6. fAsym: continuous # distance from highest pixel to center, projected onto major axis [mm] \n",
    "7. fM3Long: continuous # 3rd root of third moment along major axis [mm] \n",
    "8. fM3Trans: continuous # 3rd root of third moment along minor axis [mm] \n",
    "9. fAlpha: continuous # angle of major axis with vector to origin [deg] \n",
    "10. fDist: continuous # distance from origin to center of ellipse [mm] \n",
    "11. class: g,h # gamma (signal), hadron (background) \n",
    "\n",
    "g = gamma (signal): 12332 \n",
    "h = hadron (background): 6688 \n",
    "\n",
    "For technical reasons, the number of h events is underestimated. In the real data, the h class represents the majority of the events. \n",
    "\n",
    "The simple classification accuracy is not meaningful for this data, since classifying a background event as signal is worse than classifying a signal event as background. For comparison of different classifiers an ROC curve has to be used. The relevant points on this curve are those, where the probability of accepting a background event as signal is below one of the following thresholds: 0.01, 0.02, 0.05, 0.1, 0.2 depending on the required quality of the sample of the accepted events for different experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "source": [
    "# Import required libraries\n",
    "from tpot import TPOTClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd \n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flength</th>\n",
       "      <th>Fwidth</th>\n",
       "      <th>Fsize</th>\n",
       "      <th>Fconc</th>\n",
       "      <th>Fconc1</th>\n",
       "      <th>Fasym</th>\n",
       "      <th>Fm3long</th>\n",
       "      <th>Fm3trans</th>\n",
       "      <th>Falpha</th>\n",
       "      <th>Fdist</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>28.7967</td>\n",
       "      <td>16.0021</td>\n",
       "      <td>2.6449</td>\n",
       "      <td>0.3918</td>\n",
       "      <td>0.1982</td>\n",
       "      <td>27.7004</td>\n",
       "      <td>22.0110</td>\n",
       "      <td>-8.2027</td>\n",
       "      <td>40.0920</td>\n",
       "      <td>81.8828</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>31.6036</td>\n",
       "      <td>11.7235</td>\n",
       "      <td>2.5185</td>\n",
       "      <td>0.5303</td>\n",
       "      <td>0.3773</td>\n",
       "      <td>26.2722</td>\n",
       "      <td>23.8238</td>\n",
       "      <td>-9.9574</td>\n",
       "      <td>6.3609</td>\n",
       "      <td>205.2610</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>162.0520</td>\n",
       "      <td>136.0310</td>\n",
       "      <td>4.0612</td>\n",
       "      <td>0.0374</td>\n",
       "      <td>0.0187</td>\n",
       "      <td>116.7410</td>\n",
       "      <td>-64.8580</td>\n",
       "      <td>-45.2160</td>\n",
       "      <td>76.9600</td>\n",
       "      <td>256.7880</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>23.8172</td>\n",
       "      <td>9.5728</td>\n",
       "      <td>2.3385</td>\n",
       "      <td>0.6147</td>\n",
       "      <td>0.3922</td>\n",
       "      <td>27.2107</td>\n",
       "      <td>-6.4633</td>\n",
       "      <td>-7.1513</td>\n",
       "      <td>10.4490</td>\n",
       "      <td>116.7370</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>75.1362</td>\n",
       "      <td>30.9205</td>\n",
       "      <td>3.1611</td>\n",
       "      <td>0.3168</td>\n",
       "      <td>0.1832</td>\n",
       "      <td>-5.5277</td>\n",
       "      <td>28.5525</td>\n",
       "      <td>21.8393</td>\n",
       "      <td>4.6480</td>\n",
       "      <td>356.4620</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Flength    Fwidth   Fsize   Fconc  Fconc1     Fasym  Fm3long  Fm3trans  \\\n",
       "0   28.7967   16.0021  2.6449  0.3918  0.1982   27.7004  22.0110   -8.2027   \n",
       "1   31.6036   11.7235  2.5185  0.5303  0.3773   26.2722  23.8238   -9.9574   \n",
       "2  162.0520  136.0310  4.0612  0.0374  0.0187  116.7410 -64.8580  -45.2160   \n",
       "3   23.8172    9.5728  2.3385  0.6147  0.3922   27.2107  -6.4633   -7.1513   \n",
       "4   75.1362   30.9205  3.1611  0.3168  0.1832   -5.5277  28.5525   21.8393   \n",
       "\n",
       "    Falpha     Fdist Class  \n",
       "0  40.0920   81.8828     g  \n",
       "1   6.3609  205.2610     g  \n",
       "2  76.9600  256.7880     g  \n",
       "3  10.4490  116.7370     g  \n",
       "4   4.6480  356.4620     g  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Load the data\n",
    "telescope=pd.read_csv('MAGIC Gamma Telescope Data.csv')\n",
    "telescope.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can be seen in the above the class object is organized here, and hence for better results, I start with randomly shuffling the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flength</th>\n",
       "      <th>Fwidth</th>\n",
       "      <th>Fsize</th>\n",
       "      <th>Fconc</th>\n",
       "      <th>Fconc1</th>\n",
       "      <th>Fasym</th>\n",
       "      <th>Fm3long</th>\n",
       "      <th>Fm3trans</th>\n",
       "      <th>Falpha</th>\n",
       "      <th>Fdist</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10207</th>\n",
       "      <td>25.4589</td>\n",
       "      <td>13.0288</td>\n",
       "      <td>2.3314</td>\n",
       "      <td>0.5175</td>\n",
       "      <td>0.3147</td>\n",
       "      <td>-8.2082</td>\n",
       "      <td>-13.0703</td>\n",
       "      <td>-12.8604</td>\n",
       "      <td>14.9630</td>\n",
       "      <td>111.5380</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9739</th>\n",
       "      <td>61.6277</td>\n",
       "      <td>31.6442</td>\n",
       "      <td>3.4755</td>\n",
       "      <td>0.1559</td>\n",
       "      <td>0.0801</td>\n",
       "      <td>53.9654</td>\n",
       "      <td>50.3227</td>\n",
       "      <td>20.6942</td>\n",
       "      <td>1.7327</td>\n",
       "      <td>280.4550</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1104</th>\n",
       "      <td>97.7331</td>\n",
       "      <td>34.3795</td>\n",
       "      <td>3.2945</td>\n",
       "      <td>0.2548</td>\n",
       "      <td>0.1546</td>\n",
       "      <td>30.8789</td>\n",
       "      <td>-48.7222</td>\n",
       "      <td>22.7980</td>\n",
       "      <td>0.9080</td>\n",
       "      <td>391.9120</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14033</th>\n",
       "      <td>26.5854</td>\n",
       "      <td>12.0168</td>\n",
       "      <td>2.6037</td>\n",
       "      <td>0.4907</td>\n",
       "      <td>0.2852</td>\n",
       "      <td>-17.7408</td>\n",
       "      <td>13.8107</td>\n",
       "      <td>5.1700</td>\n",
       "      <td>66.8370</td>\n",
       "      <td>258.3600</td>\n",
       "      <td>h</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18284</th>\n",
       "      <td>38.6553</td>\n",
       "      <td>27.6317</td>\n",
       "      <td>2.7836</td>\n",
       "      <td>0.3024</td>\n",
       "      <td>0.1583</td>\n",
       "      <td>45.9195</td>\n",
       "      <td>52.9885</td>\n",
       "      <td>-32.5781</td>\n",
       "      <td>55.6531</td>\n",
       "      <td>204.6782</td>\n",
       "      <td>h</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Flength   Fwidth   Fsize   Fconc  Fconc1    Fasym  Fm3long  Fm3trans  \\\n",
       "10207  25.4589  13.0288  2.3314  0.5175  0.3147  -8.2082 -13.0703  -12.8604   \n",
       "9739   61.6277  31.6442  3.4755  0.1559  0.0801  53.9654  50.3227   20.6942   \n",
       "1104   97.7331  34.3795  3.2945  0.2548  0.1546  30.8789 -48.7222   22.7980   \n",
       "14033  26.5854  12.0168  2.6037  0.4907  0.2852 -17.7408  13.8107    5.1700   \n",
       "18284  38.6553  27.6317  2.7836  0.3024  0.1583  45.9195  52.9885  -32.5781   \n",
       "\n",
       "        Falpha     Fdist Class  \n",
       "10207  14.9630  111.5380     g  \n",
       "9739    1.7327  280.4550     g  \n",
       "1104    0.9080  391.9120     g  \n",
       "14033  66.8370  258.3600     h  \n",
       "18284  55.6531  204.6782     h  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "telescope_shuffle=telescope.iloc[np.random.permutation(len(telescope))]\n",
    "telescope_shuffle.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above also rearranges the index number and below I basically reset the Index Numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flength</th>\n",
       "      <th>Fwidth</th>\n",
       "      <th>Fsize</th>\n",
       "      <th>Fconc</th>\n",
       "      <th>Fconc1</th>\n",
       "      <th>Fasym</th>\n",
       "      <th>Fm3long</th>\n",
       "      <th>Fm3trans</th>\n",
       "      <th>Falpha</th>\n",
       "      <th>Fdist</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>25.4589</td>\n",
       "      <td>13.0288</td>\n",
       "      <td>2.3314</td>\n",
       "      <td>0.5175</td>\n",
       "      <td>0.3147</td>\n",
       "      <td>-8.2082</td>\n",
       "      <td>-13.0703</td>\n",
       "      <td>-12.8604</td>\n",
       "      <td>14.9630</td>\n",
       "      <td>111.5380</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>61.6277</td>\n",
       "      <td>31.6442</td>\n",
       "      <td>3.4755</td>\n",
       "      <td>0.1559</td>\n",
       "      <td>0.0801</td>\n",
       "      <td>53.9654</td>\n",
       "      <td>50.3227</td>\n",
       "      <td>20.6942</td>\n",
       "      <td>1.7327</td>\n",
       "      <td>280.4550</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>97.7331</td>\n",
       "      <td>34.3795</td>\n",
       "      <td>3.2945</td>\n",
       "      <td>0.2548</td>\n",
       "      <td>0.1546</td>\n",
       "      <td>30.8789</td>\n",
       "      <td>-48.7222</td>\n",
       "      <td>22.7980</td>\n",
       "      <td>0.9080</td>\n",
       "      <td>391.9120</td>\n",
       "      <td>g</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>26.5854</td>\n",
       "      <td>12.0168</td>\n",
       "      <td>2.6037</td>\n",
       "      <td>0.4907</td>\n",
       "      <td>0.2852</td>\n",
       "      <td>-17.7408</td>\n",
       "      <td>13.8107</td>\n",
       "      <td>5.1700</td>\n",
       "      <td>66.8370</td>\n",
       "      <td>258.3600</td>\n",
       "      <td>h</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>38.6553</td>\n",
       "      <td>27.6317</td>\n",
       "      <td>2.7836</td>\n",
       "      <td>0.3024</td>\n",
       "      <td>0.1583</td>\n",
       "      <td>45.9195</td>\n",
       "      <td>52.9885</td>\n",
       "      <td>-32.5781</td>\n",
       "      <td>55.6531</td>\n",
       "      <td>204.6782</td>\n",
       "      <td>h</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Flength   Fwidth   Fsize   Fconc  Fconc1    Fasym  Fm3long  Fm3trans  \\\n",
       "0  25.4589  13.0288  2.3314  0.5175  0.3147  -8.2082 -13.0703  -12.8604   \n",
       "1  61.6277  31.6442  3.4755  0.1559  0.0801  53.9654  50.3227   20.6942   \n",
       "2  97.7331  34.3795  3.2945  0.2548  0.1546  30.8789 -48.7222   22.7980   \n",
       "3  26.5854  12.0168  2.6037  0.4907  0.2852 -17.7408  13.8107    5.1700   \n",
       "4  38.6553  27.6317  2.7836  0.3024  0.1583  45.9195  52.9885  -32.5781   \n",
       "\n",
       "    Falpha     Fdist Class  \n",
       "0  14.9630  111.5380     g  \n",
       "1   1.7327  280.4550     g  \n",
       "2   0.9080  391.9120     g  \n",
       "3  66.8370  258.3600     h  \n",
       "4  55.6531  204.6782     h  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tele=telescope_shuffle.reset_index(drop=True)\n",
    "tele.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pre-Processing Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Flength     float64\n",
       "Fwidth      float64\n",
       "Fsize       float64\n",
       "Fconc       float64\n",
       "Fconc1      float64\n",
       "Fasym       float64\n",
       "Fm3long     float64\n",
       "Fm3trans    float64\n",
       "Falpha      float64\n",
       "Fdist       float64\n",
       "Class        object\n",
       "dtype: object"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check the Data Type\n",
    "tele.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Class is a Categorical Variable as can be seen from above. The levels in it are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Levels for catgeory 'Class': ['g' 'h']\n"
     ]
    }
   ],
   "source": [
    "for cat in ['Class']:\n",
    "    print(\"Levels for catgeory '{0}': {1}\".format(cat, tele[cat].unique()))   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, the categorical variable is numerically encoded since TPOT for now cannot handle categorical variables. 'g' is coded as 0 and 'h' as 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tele['Class']=tele['Class'].map({'g':0,'h':1})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A check is performed to see if there are any missing values and code then accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Flength     False\n",
       "Fwidth      False\n",
       "Fsize       False\n",
       "Fconc       False\n",
       "Fconc1      False\n",
       "Fasym       False\n",
       "Fm3long     False\n",
       "Fm3trans    False\n",
       "Falpha      False\n",
       "Fdist       False\n",
       "Class       False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tele = tele.fillna(-999)\n",
    "pd.isnull(tele).any()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(19020, 11)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tele.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we store the class labels, which we need to predict, in a separate variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tele_class = tele['Class'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Analysis using TPOT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To begin our analysis, we need to divide our training data into training and validation sets. The validation set is just to give us an idea of the test set error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(14265, 4755)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_indices, validation_indices = training_indices, testing_indices = train_test_split(tele.index, stratify = tele_class, train_size=0.75, test_size=0.25)\n",
    "training_indices.size, validation_indices.size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After that, we proceed to calling the `fit()`, `score()` and `export()` functions on our training dataset.\n",
    "An important TPOT parameter to set is the number of generations (via the `generations` kwarg). Since our aim is to just illustrate the use of TPOT, we assume the default setting of 100 generations, whilst bounding the total running time via the `max_time_mins` kwarg (which may, essentially, override the former setting). Further, we enable control for the maximum amount of time allowed for optimization of a single pipeline, via `max_eval_time_mins`.\n",
    "\n",
    "On a standard laptop with 4GB RAM, each generation takes approximately 5 minutes to run. Thus, for the default value of 100, without the explicit duration bound, the total run time could be roughly around 8 hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning: xgboost.XGBClassifier is not available and will not be used by TPOT.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Optimization Progress: 44pipeline [00:37,  1.10s/pipeline]                  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation 1 - Current best internal CV score: 0.843603389087\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Optimization Progress: 63pipeline [00:48,  1.44pipeline/s]                  "
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation 2 - Current best internal CV score: 0.843603389087\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Optimization Progress: 80pipeline [00:55,  2.07pipeline/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation 3 - Current best internal CV score: 0.846056430638\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Optimization Progress: 100pipeline [01:12,  1.24pipeline/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation 4 - Current best internal CV score: 0.846056430638\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Optimization Progress: 121pipeline [01:29,  1.08s/pipeline]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation 5 - Current best internal CV score: 0.846056430638\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Optimization Progress: 136pipeline [01:40,  1.78pipeline/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation 6 - Current best internal CV score: 0.853347788745\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Optimization Progress: 154pipeline [01:55,  1.12s/pipeline]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generation 7 - Current best internal CV score: 0.853347788745\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                           \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "2.02892383333 minutes have elapsed. TPOT will close down.\n",
      "TPOT closed prematurely. Will use the current best pipeline.\n",
      "\n",
      "Best pipeline: DecisionTreeClassifier(LogisticRegression(input_matrix, C=10.0, dual=False, penalty=l2), criterion=gini, max_depth=7, min_samples_leaf=5, min_samples_split=7)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TPOTClassifier(config_dict={'sklearn.ensemble.GradientBoostingClassifier': {'max_features': array([ 0.05,  0.1 ,  0.15,  0.2 ,  0.25,  0.3 ,  0.35,  0.4 ,  0.45,\n",
       "        0.5 ,  0.55,  0.6 ,  0.65,  0.7 ,  0.75,  0.8 ,  0.85,  0.9 ,\n",
       "        0.95,  1.  ]), 'learning_rate': [0.001, 0.01, 0.1, 0.5, 1.0], 'min_samples_... 0.7 ,  0.75,  0.8 ,  0.85,  0.9 ,\n",
       "        0.95,  1.  ])}, 'sklearn.preprocessing.RobustScaler': {}},\n",
       "        crossover_rate=0.1, cv=5, disable_update_check=False,\n",
       "        early_stop=None, generations=1000000, max_eval_time_mins=0.04,\n",
       "        max_time_mins=2, mutation_rate=0.9, n_jobs=1, offspring_size=15,\n",
       "        periodic_checkpoint_folder=None, population_size=15,\n",
       "        random_state=None, scoring=None, subsample=1.0, verbosity=2,\n",
       "        warm_start=False)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tpot = TPOTClassifier(verbosity=2, max_time_mins=2, max_eval_time_mins=0.04, population_size=15)\n",
    "tpot.fit(tele.drop('Class',axis=1).loc[training_indices].values, tele.loc[training_indices,'Class'].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above, 7 generations were computed, each giving the training efficiency of fitting model on the training set. As evident, the best pipeline is the one that has the CV score of 85.335%. The model that produces this result is pipeline, consisting of a logistic regression that adds synthetic features to the input data, which then get utilized by a decision tree classifier to form the final predictions.\n",
    "\n",
    "Next, the test error is computed for validation purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.85573080967402737"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tpot.score(tele.drop('Class',axis=1).loc[validation_indices].values, tele.loc[validation_indices, 'Class'].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "As can be seen, the test accuracy is 85.573%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tpot.export('tpot_MAGIC_Gamma_Telescope_pipeline.py')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# %load tpot_MAGIC_Gamma_Telescope_pipeline.py\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import make_pipeline, make_union\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from tpot.builtins import StackingEstimator\n",
    "\n",
    "# NOTE: Make sure that the outcome column is labeled 'target' in the data file\n",
    "tpot_data = pd.read_csv('PATH/TO/DATA/FILE', sep='COLUMN_SEPARATOR', dtype=np.float64)\n",
    "features = tpot_data.drop('target', axis=1)\n",
    "training_features, testing_features, training_target, testing_target = \\\n",
    "            train_test_split(features, tpot_data['target'], random_state=None)\n",
    "\n",
    "# Average CV score on the training set was:0.853347788745\n",
    "exported_pipeline = make_pipeline(\n",
    "    StackingEstimator(estimator=LogisticRegression(C=10.0, dual=False, penalty=\"l2\")),\n",
    "    DecisionTreeClassifier(criterion=\"gini\", max_depth=7, min_samples_leaf=5, min_samples_split=7)\n",
    ")\n",
    "\n",
    "exported_pipeline.fit(training_features, training_target)\n",
    "results = exported_pipeline.predict(testing_features)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
